nvidia-peer-memory: prefer in-tree nvidia_peermem over legacy nv_peer_mem - #1390
nvidia-peer-memory: prefer in-tree nvidia_peermem over legacy nv_peer_mem#1390100-JM wants to merge 3 commits into
Conversation
…_mem Driver R470.42.01 and later ship nvidia_peermem as part of the driver; the out-of-tree nv_peer_mem DKMS module does not exist there. The role unconditionally ran dkms autoinstall + modprobe nv_peer_mem on DGX systems and failed on current DGX OS. Probe for the in-tree module with modinfo and load it when present; fall back to the legacy nv_peer_mem path only when it is absent. Observed on DGX OS 7.5.0 / driver 580.126.20: modprobe: FATAL: Module nv_peer_mem not found in directory /lib/modules/6.8.0-106-generic Signed-off-by: 백지명 <wlaud9805@gmail.com>
The probe task is guarded by the same GPU/DGX conditions as the tasks that consume its result, so nvidia_peermem_info.rc is only defined on hosts that reach it. Relying on when-list short-circuiting to avoid the undefined attribute is easy to break by reordering the conditions. Default the value to 1 (module absent) so each branch is safe to evaluate on its own. Behaviour is unchanged on every host. Signed-off-by: 백지명 <wlaud9805@gmail.com>
dholt
left a comment
dholt
left a comment
There was a problem hiding this comment.
Before loading nvidia_peermem, handle the legacy state explicitly: stop and disable the nv_peer_mem service when present, unload nv_peer_mem as safely required, and then load nvidia_peermem. The transition should also remain successful on a second Ansible run.
Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).
…eermem A host upgraded from a pre-R470 driver can still have the nv_peer_mem service enabled and the module loaded when nvidia_peermem becomes available in-tree. Stop and disable the service when it exists, unload nv_peer_mem, then load nvidia_peermem so the two never coexist. Every step is idempotent: service_facts gates the service task and modprobe only acts when the module state differs, so a second run reports ok throughout. Signed-off-by: 백지명 <wlaud9805@gmail.com>
|
Pushed 45c03ba to address this. When the in-tree module is available the role now: gathers Second run: One deliberate choice: if |
|
@dholt The requested changes are pushed (see the comment above for details) — ready for another look when you have a moment. |
2 participants
Problem
roles/nvidia-peer-memoryassumes the out-of-treenv_peer_memDKMS module on DGX systems. Since driver R470.42.01 the equivalentnvidia_peermemmodule ships in-tree with the driver andnv_peer_memis not available, so the role fails on current DGX OS:There is no variable to disable the role from
slurm-cluster.yml; only thenvidia-peer-memorytag.Reproduced on DGX OS 7.5.0, driver 580.126.20.
Fix
Probe for
nvidia_peermemwithmodinfo. When present, retire any legacy state first — stop and disablenv_peer_mem.serviceif it exists, unloadnv_peer_mem— then loadnvidia_peermem, and skip the legacy DKMS/nv_peer_mem/service path. When the in-tree module is absent, behaviour is unchanged.Every step is idempotent (
service_factsgates the service task;modprobeonly acts when the module state differs), so a second run reportsokthroughout.Verification
Failure reproduced on DGX OS 7.5.0 / driver 580.126.20. On the same system
lsmodshows the in-tree module already loaded (nvidia_peermem 16384 0, used byib_uverbs), so the new probe takes the in-tree branch and the legacy DKMS/nv_peer_mem path is skipped.